Computer-based method and system for providing active and automatic personal assistance using an automobile or a portable electronic device

ABSTRACT

A method and a system for providing personal assistance in daily activities. A method and a system for automatically and actively providing personal assistance, using a vehicle or a portable electronic device, based on detected data regarding a user and the user&#39;s environment. The method and system may include a processor, at least one sensor, an output device, a communications unit, and a database. The database may further include a memory and cloud-based database and computing. The method and system may actively and automatically provide personal assistance regarding health, exercise, diet, or nutrition. The method and system may assist the user or a health professional in health diagnosis and treatment.

BACKGROUND

1. Field

The present invention relates generally to a computer-based method ofproviding personal assistance in daily activities and more particularly,pertains to a method and a system of automatically and activelyproviding personal assistance, using a vehicle or a portable electronicdevice, based on detected data regarding the user and the user'senvironment.

2. Description of the Related Art

Computing power is ever advancing. Records of data may be made with moreease than was previously obtainable. Prior personal assistance andmemory jogging methods/systems implemented in a vehicle or a portableelectronic device have heavily relied upon passively outputting databased on an input/request from the user or upon occurrence of apredetermined or scheduled event stored in a database. However,currently, a computer-based method or system is needed for actively andautomatically providing personal assistance to the user (e.g., activelyand automatically reminding the user of an event or actively andautomatically encouraging the user to take a particular action based onconversational and situational cues). Thus, a method/system is neededfor detecting and analyzing data regarding the user and the environmentsurrounding the user, and actively and automatically providing personalassistance based on the detected and analyzed data.

SUMMARY

The present invention relates generally to a computer-based method ofproviding personal assistance in daily activities and more particularly,pertains to a method and a system of automatically and activelyproviding personal assistance, using a vehicle or a portable electronicdevice, based on detected data regarding the user and the user'senvironment.

In one embodiment, the present invention may be, for example, acomputer-based method including the steps of: detecting, using at leastone sensor, a data associated with at least one person, an object, aliving being, a place, an environment, or combinations thereof within aproximity of at least one of the at least one sensor; selectivelystoring, using a processor, a learned data in a database based on thedetected data and a pre-programmed algorithm stored in the database;passively outputting, using a communications unit or an output devicecoupled to or in communication with the processor, a first output dataupon a request received by the processor or a predetermined or scheduledevent stored in the database; and actively and automatically outputting,using the communications unit or the output device, a second output databased on the detected data, the learned data, and the pre-programmedalgorithm.

In another embodiment, the method may include analyzing, using theprocessor, the detected data based on the learned data corresponding tothe at least one person, the object, the living being, the event, theplace, the environment, or combinations thereof such that the steps ofselectively storing the database and actively and automaticallyoutputting the second output data are further based on the analyzeddata. In yet another embodiment, the method may include analyzing, usingthe processor, a conversation or an interaction based on the detecteddata, such that the step of actively and automatically outputting thesecond output data further includes at least one of: actively andautomatically generating, using the communications unit or the outputdevice, a sound, an image, a video, or combinations thereof for activelyand automatically initiating or participating in the conversation or theinteraction with the at least one person; or actively and automaticallygenerating, using the communications unit or the output device, thesound, the image, the video, or combinations thereof for activelycommunicating a reminder to the user.

In yet another embodiment, the present invention may be a vehicleincluding: at least one sensor configured to detect a data associatedwith a driver, a passenger, an object, a living being, an event, aplace, an environment, or combinations thereof within a proximity of atleast one of the at least one sensor; a database storing apre-programmed algorithm; a communications unit; and a processor coupledto or in communication with the at least one sensor, the database, andthe communications unit, the processor being configured to: selectivelystore a learned data in the database based on the detected data and thepre-programmed algorithm; analyze the detected data based on the learneddata and the pre-programmed algorithm; passively outputting, using thecommunications unit, a first output data upon a request received by theprocessor or a predetermined or scheduled event stored in the database;and actively and automatically output, using the communications unit, asecond output data based on the analyzed data, the learned data, and thepre-programmed algorithm.

In yet another embodiment, the present invention may be a portableelectronic device including: at least one sensor configured to detect adata associated with at least one person or a user, an object, a livingbeing, an event, a place, an environment, or combinations thereof withina proximity of at least one of the at least one sensor; a databasestoring a pre-programmed algorithm; a communications unit; and aprocessor coupled to or in communication with the at least one sensor,the database, and the communications unit, the processor beingconfigured to: selectively store a learned data in the database based onthe detected data and the pre-programmed algorithm; analyze the detecteddata based on the learned data and the pre-programmed algorithm;passively output, using the communications unit, a first output dataupon a request received by the processor or a predetermined or scheduledevent stored in the database; and actively and automatically output,using the communications unit, a second output data based on theanalyzed data, the learned data, and the pre-programmed algorithm.

BRIEF DESCRIPTION OF THE DRAWINGS

Other systems, methods, features, and advantages of the presentinvention will be or will become apparent to one with skill in the artupon examination of the following figures and detailed description. Itis intended that all such additional systems, methods, features, andadvantages be included within this description, be within the scope ofthe present invention, and be protected by the accompanying claims.Component parts shown in the drawings are not necessarily to scale, andmay be exaggerated to better illustrate the important features of thepresent invention. In the drawings, like reference numerals designatelike parts throughout the different views, wherein:

FIG. 1 is a block diagram of a computer-based method/system ofautomatically providing personal assistance according to an embodimentof the present invention;

FIGS. 2A and 2B illustrate a computer-based method/system implemented ina vehicle for detecting and processing conversations and events, andactively and automatically providing guidance to the user according toan embodiment of the present invention;

FIG. 3 illustrates a portable electronic device actively andautomatically outputting data for providing personal assistance to auser according to an embodiment of the present invention;

FIG. 4 illustrates a decision flowchart diagram showing a computer-basedmethod/system of outputting a first and a second output data based ondetected data according to an embodiment of the present invention;

FIG. 5 illustrates a computer-based method/system of actively andautomatically providing personal assistance to a user using a portableelectronic device according to an embodiment of the present invention;

FIGS. 6A and 6B illustrate a computer-based method/system for providingpersonal assistance regarding health concerns to a user using a portableelectronic device according to an embodiment of the present invention;

FIG. 7 illustrates a computer-based method/system for actively andautomatically analyzing and storing detected data according to anembodiment of the present invention;

FIG. 8 illustrates a computer-based method/system implemented in aportable electronic device of actively and automatically outputting asecond output data for providing personal assistance regarding healthconcerns to a user according to an embodiment of the present invention;

FIG. 9 illustrates a computer-based method/system of actively andautomatically providing personal assistance to a user with memoryretention/retrieval difficulties using a portable electronic deviceaccording to an embodiment of the present invention;

FIG. 10 illustrates a computer-based method/system of providing personalassistance using a two-wheeled balancing device according to anembodiment of the present invention; and

FIG. 11 illustrates a computer-based method/system of providing personalassistance using a wheelchair or a three-wheeled transportation deviceaccording to an embodiment of the present invention.

DETAILED DESCRIPTION

Referring to FIG. 1, a block diagram is shown of a method/system 100according to an embodiment of the present invention. The method/system100 may include a processor 102, a communications unit 104, sensors 106,a database 112, and an output device 114. The database 112 may include amemory 108 and cloud-based database and computing 110. In otherembodiments, the database 112 may include solely the memory 108 orsolely the cloud-based database and computing 110. The various units ofthe method/system 100 may be in communication with one another by usingtransmission of an electronic signal through a Control Area Network(CAN) bus. In other embodiments, the control and communications may beover various other types of serial communication links, direct wirings,digital communication buses, wireless communications, or othercommunication links and networks.

In general, in the method/system 100, the database 112 stores apre-programmed algorithm. The method/system 100 continuously detects adata associated with a user or a person within a proximity of at leastone of the sensors 106. The method/system 100 may further detect dataassociated with a static or dynamic object, a living being, asurrounding environment/place, and other detectable matters within theproximity of at least one of the sensors 106. The method/system 100 notonly outputs a first output data (using the communications unit 104and/or using the output device 114) upon a request received by theprocessor 102 or a predetermined event stored in the database, but alsoactively and automatically outputs a second output data (via thecommunications unit 104 and/or using the output device 114). As such,the method/system 100 is configured to actively and automaticallyprovide personal assistance to the user by detecting data regarding theuser and the surrounding environment of the user, analyzing, storing,and cataloging the data, and outputting a second output data which can,for example, include helpful information as needed and as appropriateunder the circumstances as determined by appropriateness algorithm rulesdescribed below (including but not limited to privacy concern rules).The processor 102 may modify or supplement the appropriateness algorithmrules based on the learned data. Analyzing and variants thereof, as usedherein, may refer to including, but not limited to, any analysis,parsing, cataloging, or any other type of processing performed usingdetected data that may assist in storing data for later use, analyzingdata to draw an inference, retrieving data, or outputting data.

The method/system 100 is not limited to any particular device, unit, orplatform. In another embodiment, the method/system 100 is integrated ina vehicle. A vehicle as used herein includes, but is not limited to, anytype of transportation device such as an automobile, a wheelchair, awalker, a two-wheeled self-balancing vehicle, a scooter, a three-wheeledvehicle, or various other devices which may enable or assist a user intransportation.

In an embodiment illustrated in FIGS. 2A and 2B, a computer-basedmethod/system implemented in a vehicle for detecting and analyzingconversations and events, and actively and automatically providingassistance and guidance to the user, according to another embodiment ofthe present invention. The processor 102 may be an Electronic ControlUnit (ECU) of the vehicle or any other processor capable of processingdata. The memory 108 may be employed in the vehicle and coupled to or incommunication with the processor 102. The memory 108 may be, forexample, a random access memory (RAM) or a read only memory (ROM). TheECU or a separate processor may have access to the Internet or be incommunication with the cloud-based database and computing 110. Variouscloud-based solutions may be utilized without limiting the scope of theinvention. The communications unit 104 or the output device 114 may bean input/output interface 210 illustrated in FIG. 2A. The sensors 106may be located at sensor locations 220 in order to detect data.

Referring to FIG. 1, in yet another embodiment, the method/system 100may be a software program or application (“app”) implemented in aportable electronic device. A portable electronic device as used hereinincludes, but is not limited to, a laptop, a mobile communication devicesuch as a cellular phone or a smart phone, a tablet, a handheld PC, aPDA, a smartbook, and other portable electrical or electromechanicaldevices capable of processing data.

Referring to FIG. 1, in one embodiment, method/system 100 may utilizethe sensors 106 to detect data within a proximity of the user, therebyallowing the processor 102 to draw inferences regarding the surroundingenvironment of the user. The processor 102 may then analyze the detecteddata in order to extract information and draw inferences that may enablethe method/system 100 to assist the user in real time or store theanalyzed data for automatically providing personal assistance at a latertime. Such detected data may be related to persons, objects, events,places or any other information that the sensors 106 may detect in orderto enable the processor 102 to draw inferences regarding any matter inan environment within a proximity of at least one of the sensors 106.

The sensors 106 may include converters that measure a physical quantityand convert the measured physical quantity into a signal which can beanalyzed or processed by the processor 102. The sensors 106 may be anytype of sensor and/or a plurality of sensors operating in concerttogether. The sensors 106 may be integrated in or coupled to themethod/system 100. The sensors 106 may be positioned at a distance awayfrom the method/system 100 such that the sensors 106 are incommunication with the processor 102, e.g., via a network. The processor102 may be coupled to or in communication with external devices in orderto supplement the data detected by the sensors 106. The sensors 106 maybe integrated in or coupled to an input/output communications interface,the output device 114, or various other devices or platforms withoutlimiting the scope of the invention. For example, the input/outputcommunications interface may be integrated in or coupled to a vehicle ora portable electronic device.

Referring to FIGS. 2A and 2B, in the method/system 200 implemented in avehicle, the sensors 106 may be positioned anywhere in the interior orthe exterior of the vehicle without limiting the scope of the invention.For example, the sensors 106 may be positioned at sensor locations 220shown in FIG. 2A and/or sensor locations 230 shown in FIG. 2B. Thesensors 106 may alternatively or in addition, be implemented in anexternal device in communication with or coupled to the vehicle.

Similarly, in an embodiment in which the method/system 100 isimplemented in a portable electronic device, the sensors 106 may beintegrated in the portable electronic device. The sensors 106 mayalternatively or in addition, be implemented in an external device incommunication with or coupled to the portable electronic device.

The detected data, as used herein, includes, but is not limited to, anydetected data that may assist the processor 102 in drawing an inferenceregarding a user or a person. The detected data includes, but is notlimited to, a physical appearance, motion, interaction, action,behavior, conversation, characteristic, location, identity, orcombinations thereof of at least one person or a user within a proximityof the user or at least one of the plurality of the sensors 106. Thedata, as used herein, may also include any data that may assist theprocessor 102 to draw an inference regarding any matter in anenvironment within a proximity of the user or at least one of theplurality of sensors 106. The data includes, but is not limited to, acharacteristic, location, condition, type, or any attribute associatedwith an object, living being, surrounding place/environment, and othermatters therein within a proximity of at least one of the sensors 106.

Proximity as used hereinafter refers to an area around the user orwithin the detection range of at least one of the sensors 106 withinwhich data is sought to be collected. However, proximity, as usedherein, is not limited to any particular distance range as the distancemay vary depending on detection range of the sensors 106, the extent ofarea in which data is sought to be collected, and various other designor practical factors or limitations. In another embodiment, the detecteddata may not be limited to an environment within a proximity of theuser. For example, in one embodiment, GPS data may be detected regardinga location that may be positioned well beyond the detection range of allof the sensors 106. The processor 102 may analyze the detected GPS dataand automatically direct the communications unit 104 to output a secondoutput data based on the detected GPS data.

In one embodiment, the sensors 106 may include sound and speech sensorssuch as a microphone capable of detecting speech, conversation, orsound. Speech as used herein may refer to speech, voice, or any othersound conveying information. In yet another embodiment, a 3-D(three-dimensional) microphone may be utilized to detect speech andfurther pinpoint or detect the location of the source of the detectedspeech.

For example, referring to FIG. 2A, a microphone may be positionedanywhere in the passenger compartment of the vehicle without limitingthe scope of the invention. For example, the microphone may bepositioned at any of the sensor locations 220. The microphone may bepositioned, for example, on the steering wheel or an area adjacent tothe steering wheel. The microphone or an additional microphone may bepositioned, for example, in the driver or passenger overhead. Themicrophone may be utilized to detect a speech of the user 202 and/or theperson 206. The detected speech may be analyzed as described in detailsbelow with respect to FIGS. 3-7 and 9.

The sensors 106 may further include an image sensor or a camera that maybe integrated in, coupled to, or in communication with the method/system100 for capturing images/videos. In an embodiment implemented in aportable electronic device, a camera embedded in the portable electronicdevice may be utilized to capture an image or a video. For example, ifthe method/system 100 is implemented in a smartphone, mobileapplications may be operated to add capabilities to the camera or toprocess the image or video captured by the camera.

A camera embedded in a portable electronic device may have limitedvantage points or a blocked view at times. In one embodiment, when theuser is indoors, a camera may be utilized as an external device placedin a stationary position that has a better vantage point for capturingand sensing data. The camera may be in communication with the processor102, thereby transmitting the detected data to the processor 102. Forexample, the camera may supplement the detected data regarding the atleast one person or user by capturing a visual facial feature of theuser/person.

Referring to FIGS. 2A and 2B, the camera may be positioned, for example,at the sensor locations 220. For example, the camera may capture animage or a video of the user 202 and/or the person 206. In addition oralternatively, a camera may be positioned on the exterior of the vehicleat sensor locations 230 for detecting data regarding an object, a livingbeing, an event, a place, an environment (e.g., data regarding steepgrade or vibration), or combinations thereof within a proximity of thevehicle. During the normal course of transit of the vehicle, data may bedetected using the sensors 106 regarding the user 202 or the person 206and stored in the database 112 for automatically providing personalassistance in real time or at a later time.

The sensors 106 may include positional encoders, compasses,navigational, and GPS sensors. The method/system 100 may be incommunication with or coupled to a GPS system for supplementing thedetected data. For example, if the method/system 100 is mounted on amobile unit (e.g., a vehicle/transportation device), the sensors 106 mayinclude an inertial measurement unit (IMU), which detects velocity,orientation, and gravitational forces of the mobile unit, using acombination of accelerometers, compasses, distance sensors, geomagneticsensors, and gyroscopes. The sensors 106 may include variousproximity/position sensors.

The sensors 106 may further include a capacitive displacement sensor, apassive thermal infrared sensor, a photocell (reflective) sensor, aradar sensor, a vibration sensor, a sonar sensor, and/or the like. Thesensors 106 may also be configured to provide a user's current locationand identify objects in the vicinity and/or within a geographic range.The sensors 106 may include perimeter monitoring sensors or ultraviolet,radio frequency, or infrared sensors. A three dimensional (3-D) scannermay be employed to analyze a real-world object or environment to collectdata on its shape and its appearance (e.g., color). The detected data isthen communicated to the processor 102 for constructing digital 3-Dmodels.

The sensors 106 may further detect atmospheric pressure, smoke, andvarious other attributes of the environment within a proximity of theuser. The sensors 106 may utilize thermistors to measure temperature.Based on the detected data, the processor 102 may determine the type ofthe environment (e.g., a shopping mall, a parking lot, an office, andother place/environment characteristics). For example, the processor 102may further determine the current season or weather and the currenttemperature or moisture content based on the detected data. The sensors106 may further include tactile sensors utilized to analyze contact withan object, person, living being, and other matters therein. In oneembodiment, a touch sensor and/or a force sensor may be utilized tosupplement the tactile information.

The sensors 106 may include sensors capable of detecting odor, magneticfields, or radiation. Various other sensors may be integrated in,coupled to, or in communication with the sensors 106 in order to improvethe quality or increase the quantity of the detected data withoutlimiting the scope of the present invention.

Prior to a further detailed description of various embodiments of themethod/system 100, an example of an application of the method/system 100is provided.

Referring to FIG. 3, the method/system 300 serves as a personalassistant in daily activities. When the method/system 300 recognizesthat the user 302 is searching for an object, i.e. a key 314, theportable electronic device 304 directs the user 302 to the key 314 bylooking up prior relevant learned data and inferring context andinteraction points. In an embodiment, the method/system 300 recognizesthat the user 302 is searching for the key 314 and draws an inference310 that the key 314 was placed on the table 312, by looking up priorrelevant learned data. The method/system 300 directs the output device114 (e.g., a speaker) to output a second output data corresponding torelevant helpful information regarding the location of the key 314. Theoutput device 114 may be integral to, coupled to, or in communicationwith the method/system 300.

Outputting relevant helpful information includes outputting a firstoutput data upon an input/request from the user or upon occurrence of apredetermined/scheduled event, and actively and automatically outputtinga second output data based on the detected data and further based on alearned data. The learned data as used herein refers to currently orpreviously detected, processed, and stored data. The processor 102 maydirect the portable electronic device 304 to passively output a firstoutput data when the processor 102 receives, using the sensors 106, ordetermines via the detected data a request by the user 302 inquiringabout the location of the key 314. The input/request may be receivedvia: recognition of a facial expression detected by a camera andanalyzed by the processor 102; input/request received from the user 302using a communication interface of the portable electronic device 304;an input/request from external devices coupled to or in communicationwith the portable electronic device 304; other devices and methods ofdetecting an input/request; or combinations thereof.

However, an important aspect of the invention is that the method/system300 is further configured to automatically output a second output dataeven without an input/request received from the user 302 and withoutoccurrence of a predetermined or scheduled event. Recognition of thefact that the user 302 is searching for the key 314 can be performedusing a combination of the sensors 106 described above. Themethod/system 300 is not limited to a particular sensor of the sensors106 or any particular means of detecting such data.

In one embodiment, for example, if the user 302 utters a speech/thought306, the sensors 106 detect the speech/thought 306. The processor 102 ofthe portable electronic device 304 can analyze the speech/thought 306 byparsing the elements of the speech/thought 306 to determine that theuser 302 is attempting to search for the key 314. Additionally oralternatively, a camera may be integrated in or coupled to the portableelectronic device 304. Alternatively or in addition, the portableelectronic device 304 may detect a facial expression of the user 302indicating a sense of confusion as to the current location of the key314. Other sensed data such as tactile data may be detected as well. Bylooking up prior relevant learned data about the key 314, themethod/system 300 matches speech elements with the object (key 314) andactions, place, time, and the like associated with the object (key 314).The processor 102 of the method/system 300 directs the portableelectronic device 304 to output a second output data corresponding torelevant and appropriate information for assisting the user 302 infinding the key 314.

For example, the portable electronic device 304 may generate an outputreminder/speech 308 indicating to the user 302 that the key 314 isplaced on the table 312. The portable electronic device 304 may,alternatively or in addition, display an output video recorded (orrecording) of the key 314 or the action of the user 302 placing the key314 on the table 312. In one embodiment, the user 302 is only informedof the location of the object (e.g., the key 314).

For example, using the sensors 106, data is continuously orintermittently detected regarding a user 302 or a person during dailyactivities. The sensors 106 further continuously or intermittentlydetect data regarding static or dynamic objects, living beings,place/environment, or combinations thereof within a proximity of thesensors 106. During daily activities of the user 302, the method/system300 may collect data associated with actions, interaction, speech,conversation, location, and other characteristics of the user 302 andthe surrounding environment within a proximity of the user 302.

For example, the method/system 300 may have recognized, using thesensors 106 and the processor 102, that the key 314 was being placed onthe table 312 based on a pre-programmed algorithm and previouslydetected, processed, and stored data as discussed in detail below. Giventhe detected data regarding a person, the user 302, objects (e.g., thekey 314 or the table 312), places (e.g., the home environment around theuser 302), the speech/thought 306, other detected or stored data, orcombinations thereof, the processor 102 of the method/system 300 caninfer context and key interaction points or events. For example, a keyinteraction point in the embodiment shown in FIG. 3, may have been theevent of “<person> placing <object1> on <object2>.” The <person>,<object1>, and <object2> may correspond to the user 302, the key 314,and the table 312, respectively. The event of “<person> placing<object1> on <object2>” may be inferred from a sequence of detectedevents as follows: “<person> holding and/or touching <object1>,”“<object1> touching and positioned above <object2>”, “<person> at adistance from or not touching <object1>.” Similarly, picking up anobject may be recognized. For example, the event “<person> picking up<object1>” can be composed of a sequence such as “<object1> next to<person>,” “<person> touches <object1>,” and “<person> holding<object1>.” As such, the method/system 300 may utilize algorithms of thelike to automatically detect and analyze relatively high level events byrecognizing a sequence of relatively lower level events. The algorithmsmay be adjusted or developed as the method/system 300 collects more dataregarding the user 302 and the matters within the surroundingenvironment of the user 302.

The recognition of placement of the key 314 on the table 312 may havebeen via analyzing a conversation or speech indicating placement of thekey 314 on the table 312. The recognition may alternatively or inaddition be based on capturing image/video of the placement of the key314. Upon using such a process for recognizing the various events withinthe surrounding environment of the user 302, the method/system 300 cancreate a journal summary of daily life of the user 302.

The method/system 300 may be applied in various applications and withrespect to various objects and events to provide personal assistance tothe user 302. For example, in an alternative embodiment, themethod/system 300 may determine based on the combinations of dataincluded in the detected data whether the user 302 is thirsty orforgetful of where a bottle of water has been placed. The processor 102of the portable electronic device 304 can direct the communications unit104 to output a second output data containing information as to thelocation of the water bottle.

Referring to a flowchart of a method/system 400 shown in FIG. 4, in step404, data is detected, using the sensors 106 as described above withrespect to FIGS. 1-3. The method/system 400 is only an example of thedetection and analysis of the data. The steps of the method/system 400may be performed simultaneously or in various combinations of orders.The degree to which the data can be collected regarding the surroundingenvironment of the user and matters therein may depend on what or whichsensors 106 are available and the processing limitations of theprocessor 102 and/or the cloud-based database and computing 110. Assuch, the method/system 400 may be adjusted accordingly in real time bymonitoring such limitations.

The processor 102 may work in concert with the sensors 106 for improvingcollection of the data. The processor 102 may also consider whether theuser or a person is requesting or attempting to convey information. Forexample, if a user is making a facial expression without speech tocommunicate with the method/system 400, the processor 102 can direct thecommunications unit 104 to pose follow-up questions or inquiries usingthe communications unit 104 in order to supplement or clarify thedetected data. For example, the method/system 400 may direct an outputspeech to be generated, thereby asking the user to clarify the facialexpression. The user may then respond in a voice command clarifying theconveyed information.

In step 406, data regarding dynamic objects, living beings, and/orplace/environment is detected and inferences are drawn accordingly. Inone embodiment, if a camera is utilized in the method/system 400 and thecamera has vantage points to detect images/videos containing usefulinformation regarding the surrounding environment of the user, themethod/system 400 may utilize the camera in addition to or as analternative to a microphone to detect the data regarding the surroundingenvironment of the user and matters therein. For example, in step 406,the method/system 400 may detect movement or changes in a scene or otherdynamic regions as observed by cameras in order to focus the sensors 106on the detected dynamic regions. The processor 102 classifies thedetected dynamic region as described below.

For example, detection of a person, living being, and/or a dynamicobject may be performed by looking for changes in data detected by thesensors 106. For example, changes in data received from the camera maybe determined. Changes in data detected by the sensors 106 may beidentified by first estimating the motion of the method/system 400 or amobile platform upon which the method/system 400 is implemented, usingGPS, IMU, or techniques such as visual odometry which allow estimationof the motion of a camera by tracking corner or blob features betweentwo camera frames. As such, the method/system 400 may identify motion inthe surrounding environment of the user which does not correspond to themotion of the method/system 400.

Upon identifying the changing parts of the scene within the detecteddata, the method/system 400 seeks to recognize the changing elements,using techniques such as “eigenfaces” and “skeletal recognition” torecognize persons and faces. Additionally, standard techniques likeSupport Vector Machines, Deformable Parts Model and dynamic programmingcan be used to learn different models for various object/person classes.The types of features that can be used in these recognition tasks can beany combination of features like SIFT (Scale-Invariant FeatureTransform), SURF (Speeded Up Robust Features), Gist modeling, Sobel,Fast, and other features and/or techniques that enable the method/system400 to recognize a person, object, living being, or place/environmentwithin a proximity of the user.

Thus, by detecting the dynamic regions, a dynamic object or being can bedetected in step 406, and a new person entering the environment withinthe proximity of the user may be detected in step 412 and classified bythe processor 102 in the database 112 accordingly. In step 406, themethod/system 400 may extract the remaining static regions of thedetected image/video in order to gather data regarding static objects ora place/environment within a proximity of the user.

The sensors 106 may utilize a microphone and a speech recognition moduleto detect speech, conversation or interaction as shown in step 408. Themethod/system 400 may further extract conversation elements containinguseful data in step 410. In step 422, the processor 102 of themethod/system 400 matches extracted conversation or speech elements fromstep 410 to the detected person, object, living being,place/environment, or combinations thereof.

In step 428, the method/system 400 looks up prior relevant informationbased on context and based on the matched conversation events from step422 regarding the person, object, living being, place/environment, orcombinations thereof. In step 426, the processor 102 stores relevantinformation for later use in the database 112 based on prior relevantinformation. For example, if the processor 102 detects facial featuresof a person entering the environment and also detects that the newperson is speaking, the speech elements can be matched with the newperson. Speech data related to the person may be stored in the database112 for later use. A 3-D microphone or a microphone array may also beutilized to localize the origin of the sound or voice. The method/system400 can track and log data related to the person in order to supplementthe detected data. The method/system 400 may actively and automaticallyoutput a second output data in step 430 based on the matchedconversation events to the corresponding person, object, living being,place/environment, or combinations thereof of step 422 and based on theinferred context and interaction key points from step 424.

The processing of data (e.g., in steps 406-430) can be performed bycontinuously analyzing data gathered by the sensors 106 in real time.The cloud-based database and computing 110 may be utilized due torestraints on the information storage capacity of the memory 108 orenergy capacity challenges associated with processing using solely theprocessor 102. However, in one embodiment, both on-board and off-boardprocessing capabilities are utilized to prepare for events in whichon-board processing may be preferable (e.g., a poor connection in cloudcommunications) to ensure a minimal level of capability. For example, ifthe method/system 400 is implemented in a portable electronic devicethat may not have sufficient capacity to perform the steps describedherein, the cloud-based database and computing 110 can provideassistance in sharing the load of processing.

In step 430, the processor 102 may passively output, using thecommunications unit 104, a first output data upon an input/requestreceived by the processor 102 or a predetermined or scheduled eventstored in the database 112.

The processor 102 may further actively and automatically output, usingthe communications unit 104, a second output data based on the detecteddata, previously detected, stored, and processed data, thepre-programmed algorithm, or combinations thereof.

For example, the first or second output data may be displayed via adisplay screen or via generation of an output speech via a speaker. Inan embodiment in which the method/system 400 is implemented in aportable electronic device, the input/output user interface, display,speakers, microphone, or combinations thereof may be utilized as thecommunications unit 104. Referring to FIG. 2A, in an embodiment in whichthe method/system 400 is implemented in a vehicle, the input/outputinterface 210, display, speakers, microphone, or combinations thereof ofthe vehicle may be utilized as the communications unit 104.

The display may be, for example, a (color) liquid crystal display onwhich the second output data is displayed. The output images/videos maybe displayed using an LCD, an organic light emitting display, a plasmadisplay, light-emitting diodes, or any other display mechanism fordisplaying the output images/videos. In another embodiment, due topractical concerns, a projector in lieu of a display screen may beutilized to project output images/videos on a wall, screen, or surfacesof the like based on the first and/or second output data. A projectormay be preferable if the method/system 400 is designed for a device withcharacteristics requiring a compact design.

The processor 102 may further submit the first and/or second output datato an output device 114 (e.g., another electronic device authorized bythe user). In another embodiment, the output device 114 is coupled to orin communication with a device including actuators and capable ofoperating mechanical movements.

In an embodiment in which the method/system 400 is implemented in avehicle as shown, for example, FIGS. 2A and 2B, a pre-programmedalgorithm may be stored in the database 112 with respect to an outputtiming (e.g., when to generate an output speech and/or when to displayan output image/video), a form (e.g., generating an output speech versusdisplaying an output video), and an information content of the outputtedfirst and/or second output data. For example, the timing and informationcontent of the actively and automatically generated output speeches ordisplayed output images/videos may be based on a determined expectedlevel of driving attention desirable for driving the vehicle under thecurrent circumstances. The determined expected level of drivingattention may depend upon a current operation of the vehicle. Forexample, when the processor 102 determines that the vehicle is inmotion, the processor 102 determines that the current expected level ofdriving attention has increased or is high. For example, the processor102 may direct the communications unit 104 to generate an output speechwhen appropriate under the circumstances based on the detected data. Onthe contrary, when the processor 102 determines that the vehicle is notin motion and/or an engine of the vehicle is not operating, theprocessor 102 determines that the current expected level of drivingattention has decreased. For example, under such circumstances, theprocessor 102 may direct the communications unit 104 to display a videoin addition to or in lieu of generating the output speech.

Referring to FIG. 5, the method/system 500, the portable electronicdevice 504 can utilize the sensors 106 (e.g., the microphone) to detectthe data regarding the conversation between the user 502 and the person506. The processor 102 can parse the conversation and store relevantelements in the database 112. An image or video of the person 506 may becaptured using a camera and analyzed by the processor 102 as describedabove with respect to steps 402-422 of FIG. 4. A first and/or secondoutput data may be outputted using the process described above withrespect to the method/system 400.

Referring to FIG. 6A, in the method/system 600, a portable electronicdevice 604 may provide personal assistance to the user 602 when visitinga health professional 606 for consultation, diagnosis, or treatment. Aclose-up view of the portable electronic device 604 is shown in FIG. 6Bin order to facilitate illustration of the display of the communicationsunit 104 for the portable electronic device 604 of FIG. 6A.

As with other social encounters described herein, for example, if theportable electronic device 604 determines that the user 602 may notrecall certain information with respect to the health professional 606,the visitation, or other associated information, the portable electronicdevice 604 may output a second output data for informing the user 602accordingly (if appropriate under the circumstances). For example, theportable electronic device 604 may output a second output data remindingthe user 602 of the number of years that the user 602 and the healthprofessional 606 have known each other and their previous encounters, ifthe portable electronic device 604 determines that outputting suchinformation would be appropriate under the circumstances.

An important aspect of the method/system 600 is that the first and/orsecond output data may be based on privacy concerns of the user 602(e.g., concerns of the user 602 regarding with whom, to what extent, andin what manner the user 602 would be comfortable in sharing suchinformation under the current circumstances). Privacy concerns may beprogrammed in the form of a pre-programmed algorithm stored in thedatabase 112 which may further be supplemented during operations of themethod/system 600 based on the detected, processed, and stored data.Such privacy concerns may be based on combinations of previouslydetected and processed data or the pre-programmed algorithm fordetermining what type of information is appropriate to be outputtedunder the circumstances. In one embodiment, individuals may bedistinguished in part by social interaction classes. For example,privacy concerns are less weighty when a conversation is with the healthprofessional 606, a family member or a therapist as discussed above. Assuch, some of the standards as to the extent of interaction with atrusted individual can be stored in pre-programmed data or algorithm,and some of the information regarding with whom and to what extent themethod/system 600 should interact can be adjusted based on the detecteddata.

In one embodiment, the method/system 600 may associate a high level oftrust with a person based on prior detected conversations of the user602 with the person. That is, a detected friend having close/intimaterelations with the user 602 can be distinguished from an acquaintance ora stranger based on prior conversations and interactions. Based in parton the recognized distinction, the portable electronic device 604 mayadjust the degree and type of information included in the second outputdata. For example, if the processor 102 determines that because the user602 would be comfortable with sharing certain information with thehealth professional 606 (acting in a professional capacity withexpectations of upholding the privacy of the user 602), the portableelectronic device 604 would output a higher degree of or more privateinformation than the portable electronic device 604 would if theinteractions were directed to an individual encountered for the firsttime and not acting in a confidential and professional capacity. Themethod/system 600 can determine whether the detected person is a healthprofessional 606, therapist, or other professionals endowed with highlevel of trust and expectation of upholding privacy of communicationsbased on the detected data, pre-programmed algorithm, and previouslystored and processed data as discussed above with respect to FIG. 4.

In one embodiment as shown in FIG. 6, for example, the healthprofessional 606 may inquire about the reason, the timing, or the causeof an arm injury sustained by the user 602 which has prompted visitingthe health professional 606 by posing the speech/inquiry 608 to the user602. The speech/inquiry 608 may be “How and when did your arm getinjured?” directed to the user 602. The portable electronic device 604may parse and analyze the conversation as described above in order toanalyze the elements of the speech/inquiry 608 and the relationship,sequence, or correlation of the elements to one another based onpreviously detected, processed, and stored data. The portable electronicdevice 604 may search the database 112 regarding previous injuriessustained. The portable electronic device 604 may match the elements of“arm” and “injury” with a slip and fall accident leading to an arminjury which was sustained during a prior week and cataloguedaccordingly. After analyzing the elements of the speech/inquiry 608, theportable electronic device 604 may determine that the healthprofessional 606 is interested in learning about the injury and thatdisclosing such information via the second output data would beappropriate in the current circumstances. The portable electronic device604 may actively and automatically output a second output data informingthe health professional 606 and the user 602 regarding the injury. Forexample, the portable electronic device 604 may generate an outputspeech/response 610 indicating that “[name of user 602] slipped lastweek landing on his arm. Here is how it happened.” The outputspeech/response 610 may further include details regarding the recordedevent (i.e. accident) in order to output a second output data forproviding proactive assistance for an improved diagnosis and/or forproviding a reminder of when and how the injury occurred.

The portable electronic device 604 may output a first output data and/ora second output data via the output device 114 or various other devicesor methods without limiting the scope of the present invention. Forexample, the portable electronic device 604 may output a second outputdata using the communications unit 104.

In one embodiment, the portable electronic device 604 may be equippedwith a display screen of displaying an output image/video 614 asdetermined by the processor 102. For example, the portable electronicdevice 604 may have previously recorded the slip and fall accidentleading to the arm injury when the accident occurred. If so, theportable electronic device 604 may output the second output data bydisplaying the output image/video 614 in order to provide assistance forthe diagnosis, treatment, or consultation. In other embodiments, thesecond output data may be communicated to another device of the user 602or the health professional 606 (e.g., another portable electronicdevice).

The source of the detected data may not necessarily be the sensors 106.That is, multiple storage units and electronic devices may operate inconcert sharing information in order to improve the quality and increasethe quantity of information on which the outputted data are based. Forexample, the portable electronic device 604 may have access to theInternet or be in communication with the cloud-based database andcomputing 110 to receive input data previously or currently stored by anexternal device (e.g., a laptop computer) of the user 602. For example,if the user 602 has previously stored data in any of the electronicdevices in communication with the cloud-based database and computing110, the portable electronic device 604 may search such previouslystored data for relevant information regarding the arm injury and outputa second output data as described above accordingly.

Referring to FIG. 7, in one embodiment of the method/system 700, theportable electronic device 704 may be the portable electronic device 604discussed above with respect to FIG. 6. For example, the healthprofessional 706 intends to provide recommendations as to courses ofaction that would improve a health condition of the user 702. Forexample, the health professional 706 may utter the speech/recommendation708, stating that “I see how you got injured. Please maintain anutritious diet including protein and vegetables, and perform weeklyexercise/physical therapy routines of A, B, and C using 10 pound freeweights.” The processor 102 of the portable electronic device 704analyzes the elements of the speech/recommendation 708. Themethod/system 700 stores and catalogues the information relating toexercise/physical therapy routines of A, B, and C using 10 pound freeweights and the portable electronic device 704 and recognizes that theexercise/physical therapy routines of A, B, and C have to be performedonce a week as instructed. The method/system 700 may automaticallyschedule reminders in the database 112 to be outputted on a weeklybasis. The method/system 700 further stores and catalogues the analyzeddata (e.g., regarding the recommended dietary regimens) in the database112.

Referring to FIG. 8, in one embodiment of the method/system 800, theportable electronic device 804 may be the portable electronic device 704discussed above with respect to FIG. 7. The portable electronic device804 may recommend, motivate, or remind the user 802 of the informationcollected as discussed above with respect to FIGS. 6 and 7. For example,the method/system 800 may analyze the detected data in order todetermine whether the user 802 is complying with the dietary andphysical therapy guidelines set forth in the speech/recommendation 708.

For example, based on the analyzed first output data, the method/system800 may determine that the user 802 has only performed physical exerciseroutine A during the past week. The method/system 800 may furtherdetermine that the user 802 is watching the TV 812 and eating a certaintype of food 814 (e.g., a cheesecake) using various object recognitiondetection and processing methods and systems. The method/system 800 mayfurther recognize that the user 802 is currently resting on the couch,and has not performed a substantial degree of physical activities duringthe past week.

The method/system 800 may search the Internet, the cloud-based databaseand computing 110, the memory 108, or other data to determine thenutritional value of the food 814. For example, the method/system 800may generate the output speech 808 in order to encourage the user 802 toundertake a healthier food choice in order to facilitate a fasterrecovery based on the recommendations of the health professional 706.The method/system 800 may further seek to motivate the user 802 toperform the physical therapy routines of B and C. The output speech 808may be “A week has passed since you last performed physical therapyroutines B and C. You can use the light weights in the room per healthprofessional's instructions. There are vegetables and eggs in the fridgeto help maintain a nutritious diet.” In another embodiment, the portableelectronic device 804 may further remind the user 802 of takingmedication properly if the health professional 706 had prescribedmedication and provided instructions for taking the medication.

In other embodiments, the method/system 800 may be coupled to or incommunication with a pedometer or other systems providing healthinformation in order to have access to additional information regardingthe health of the user 802. Alternatively, the method/system 800 maydetermine health factors based on the detected data corresponding toexercise activities, dietary habits, and other matters with or withoutan external health monitoring device.

Referring to FIG. 9, in method/system 900, the user 902 may suffer fromAlzheimer's complications or dementia, or alternatively, the user 902may have certain difficulties with retaining, retrieving, or analyzinginformation using human brain/memory. The method/system 900 mayrecognize that the user 902 is suffering from such complications basedon the pre-programmed algorithm and/or detected data. In one embodiment,the method/system 900 may implicitly provide information that isappropriate under the circumstances in order to jog the memory of theuser 902. For example, a portable electronic device 904 may becontinuously analyzing detected data within the proximity of the user902 in order to determine if a new object, person, living being, or theother matter has entered the environment.

For example, as shown in FIG. 9, the portable electronic device 904 mayrecognize, using the detected data via the sensors 106, that a personhas entered the environment by opening the door 914. The user 902 mayutter the inquiry/thought 908 of “Who is that? He looks familiar.” Ifthe user 902 does not utter the inquiry/thought 908, the method/system900 may recognize that the user 902 has not identified the newly enteredperson 906. For example, the processor 102 of the portable electronicdevice 904 may utilize a camera to operate face recognition of at leasta facial feature of the user 902 to determine that the user 902 isperplexed while attempting to identify the newly entered person 906. Inanother embodiment, the method/system 900 may automatically output thesecond output data to jog the memory of the user 902 in a subtle manneras described below. Such an embodiment may be set as a default mode ofoperation if the portable electronic device is aware that the user 902may have difficulties in retaining or retrieving information frommemory.

For example, the processor 102 of the portable electronic device 904 mayutilize a camera to operate face recognition of at least a facialfeature of the newly entered person 906. The method/system 900 mayrecognize that the newly entered person 906 is Michael, the nephew ofthe user 902. In another embodiment, the recognition of the identity ofthe user 902 may be through speech recognition and the source of thespeech (e.g., using a 3-D microphone). For example, if the newly enteredperson 906 utters a speech 910, the processor 102 may determine theidentity of the newly entered person 906 and look up prior relevant dataregarding the newly entered person 906 using the method/system 400.Other characteristics, interactions, physical appearance, or the likemay be analyzed in order to determine the identity of the newly enteredperson 906.

The processor 102 may take into account the reliability of a particularpiece of information detected by one of the sensors 106. For example,for identification or recognition of the newly entered person 906, theprocessor 102 may assign various probabilities of accurate detection orrecognition. For example, the assigned probabilities of accuratedetection or recognition may be based on the mechanism used to detectthe newly entered person 906. In one embodiment, recognition oridentification of the newly entered person 906 based on an identity codeprogrammed in an identity device (e.g., a cellular phone) of the newlyentered person 906 may not be as reliable as recognition based on afacial feature of the newly entered person 906 using a camera or speechrecognition using a microphone, given that the newly entered 906 couldbe carrying an identity device of another individual.

The assigned probabilities of accurate identification or recognition maybe further based on the circumstances under which the data is detected.For example, if a camera is obstructed causing the face of the newlyentered person 906 to be unrecognizable, the method/system 900 mayassign a relatively low probability of accurate detection or recognitionin identifying the newly entered person 906. If the obstruction isremoved and a facial recognition processing mechanism confirms theidentity of the newly entered person 906, a higher probability ofconfidence would be associated with the identification or recognition.As such, the processor 102 may identify or recognize the newly enteredperson 906 based on a combination of various instances of data detectionand further based on the corresponding probabilities of accuraterecognition or identification.

Referring to FIG. 9, when the method/system 900 determines that thenewly entered person 906 is “Michael,” a nephew of the user 902, theportable electronic device 904 outputs appropriate information toprovide indirect, subtle, and implicit reminders to the user 902. Forexample, if the method/system 900 recognizes that the newly enteredperson 906 is a close relative, the processor 102 of the portableelectronic device 904 may determine that outputting informationregarding the last time that the newly entered person 906 and the user902 have met would not be appropriate in the current circumstances.Instead, the method/system 900 may provide information in a subtlefashion to remind the user 902 that the newly entered person 906 is thenephew of the user 902. For example, the portable electronic device 904may direct the output reminder/speech 912 to the newly entered person,stating that “Hello Michael, your uncle [name of the user 902] will bewith you shortly,” thereby indirectly providing a reminder to the user902 in a subtle fashion as appropriate under the circumstances.

Alternatively or in addition, the user 902 may request the portableelectronic device 904 to generate additional information. The portableelectronic device 904 may passively output, using the communicationsunit 104, a first output data upon the request from the user 902.

Referring to the method/system 1000 of FIG. 10, in one embodiment, thevehicle 1004 may be a two-wheeled self-balancing device. The user 1002may shift his/her weight in order to direct the vehicle 1004 to maneuvertowards a desired direction. The handle bar of the vehicle 1004 may beincluded solely for providing additional comfort for the user 1002 giventhat the detected weight shift of the user 1002 using the base portion1014 of the vehicle 1004 may contain sufficient maneuvering directionsfrom the user 1002.

In one embodiment, the vehicle 1004 may communicate directly with theuser 1002. For example, the vehicle 1004 may have speech recognition andgeneration capabilities as discussed above. For example, a camera may beimplemented on the vehicle 1004 to capture first and/or data regardingthe surrounding environment of the user 1002. In another embodiment, theuser 1002 may communicate with a portable electronic device of the user1002 (e.g., the portable electronic device 304, 504, 604, 704, 804, or904 discussed above with respect to FIG. 3, 5, 6, 7, 8, or 9,respectively) in communication with or coupled to a communication portof the vehicle 1004. For example, a software application may beinstalled on a smart phone of the user 1002 in order to communicate datawith the vehicle 1004.

As shown in FIG. 10, the method/system 1000 has determined, based on thedetected, processed, and stored data that the user 1002 seeks to performphysical activities and exercises in order to maintain a healthierlifestyle. The method/system 1000 may include or be in communicationwith a GPS device. The method/system 1000 may recognize that a buildingis located on the right hand side of the user 1002. The method/system1000 may automatically obtain data regarding the building and determinethat the building is a gymnasium 1006. The data may be obtained from thecombination of data received from the GPS device and previouslydetected, processed, and stored data. The method/system 1000 may furtherdetermine the location of the home 1010 of the user 1002 and thedistance and directions from the home 1010 of the user 1002 to thegymnasium 1006. The method/system 1000 may automatically output a secondoutput data recommending that the user 1002 consider signing up at thegymnasium 1006.

The method/system 1000 may determine that based on the drivingdirections obtained from the GPS device, the route will take 20 minutesby travelling via the vehicle 1004 and will take 4 minutes by travellingvia an automobile on the road 1012. In order to facilitate the decisionmaking process, the method/system 1000 may provide further information,for example, by generating the following output recommendation/speech1008: “Since you mentioned that you seek to work out to lose weight, Ihave noticed that a gym has opened on your right recently. If you usethis device, it will take you 20 minutes and if you drive westboundusing road 1012, it will take you 4 minutes to get there. It may be agood idea to exercise in the morning before work.”

Referring to the method/system 1100 of FIG. 11, a vehicle 1104 may be awheelchair or a three-wheeled transportation device. Any of the sensors106 described above may be implemented on the vehicle 1104. For example,the sensors 106 may be installed in sensor locations 1110, 1120, 1124,and 1126. The sensors 106 may be positioned at any other portion of thevehicle 1104 without limiting the scope of the invention. For example, acamera may be positioned at the sensor location 1120 in order to collectdata regarding the surrounding environment. A camera, microphone, orother sensors 106 described above may also be installed at sensorlocation 1110. Perimeter monitoring sensors and laser scanning sensorsmay be installed as sensor locations 1124 and 1126. IMU's, gyroscopes,GPS system, and the like may further be mounted on the vehicle 1104. Thevehicle 1104 may further include a user interface 1112 to communicatewith the user 1102.

The output device 114 of vehicle 1104 may be a mechanical device havingactuators capable of maneuvering the vehicle 1104 based on a thirdoutput data outputted by the processor 102. For example, the vehicle1104 may include a joystick 1114 that allows the user 1102 to maneuverthe vehicle 1104 towards a desired direction. Alternatively or inaddition, tactile data of the user 1102 may be detected.

For example, in the case of an Alzheimer's patient or a user 1102 withshort term memory difficulties, if the user 1102 forgets a particulardestination, the method/system 1100 may guide the wheelchair and theuser 1102 to the destination. If the user 1102 is controlling themethod/system 1100, e.g., using the joystick 1114 as described above, aresponse such as a vibration may be outputted to alarm the user 1102that the user 1102 is heading the wrong way or to further guide the user1102 towards the desired destination. Vibration motors can furtherprovide feedback to the user 1102. Such vibration motors can beintegrated in or coupled to an article of manufacture carried or worn bythe user 1102. For example, a belt 1128 or a vest may be coupled to thevehicle 1104 by which a vibration may provide information to the user1102.

Using the sensors 106, the method/system 1100 may recognize that anorthopedic clinic 1106 is located on the left of the vehicle 1104 anduser 1102. Using the process described in details above (e.g., withrespect to the method/system 400 of FIG. 4), the method/system 1100 maymatch the detected data regarding the observed orthopedic clinic 1106with a previously stored event (e.g., an arm injury discussed withrespect to FIG. 6). Using the method/system 400 described above withrespect to FIG. 4, the method/system 1100 may output a second outputdata for providing a reminder data and recommending that the user 1102visits the orthopedic clinic 1106 in order to receive treatment beforethe injury exacerbates. For example, the method/system 1100 may generatean output speech 1108 as follows: “You can visit the Orthopedic Clinicon your left for the arm injury you sustained last week.”

Referring to FIGS. 1-11, one important aspect of the invention is thatinformation can be shared between all or various combinations of thedevices set forth above in FIGS. 1-12. For example, the portableelectronic device 304, 504, 604, 704, 804, or 904 discussed above withrespect to FIG. 3, 5, 6, 7, 8, or 9, respectively, along with thevehicles discussed above with respect to FIGS. 2A, 2B, 10, and 11, andthe external devices such as communication devices, cell phones,laptops, cameras, or other devices capable of communicating with orbeing coupled to the listed devices or units can share information,using the cloud-based database and computing 110 or the Internet, inorder to access, store, and process the pool of the detected, processed,and stored data.

As used herein, the term “network” includes any cloud, cloud computingsystem or electronic communications system or method which incorporateshardware and/or software components. Communication among the parties maybe accomplished through any suitable communication channels, such as,for example, a telephone network, an extranet, an intranet, Internet,point of interaction device (point of sale device, personal digitalassistant (e.g., android, iPhone®, Palm Pilot®, Blackberry®), cellularphone, kiosk, and the like), online communications, satellitecommunications, off-line communications, wireless communications,transponder communications, local area network (LAN), wide area network(WAN), virtual private network (VPN), networked or linked devices,keyboard, mouse and/or any suitable communication or data inputmodality. Specific information related to the protocols, standards, andapplication software utilized in connection with the Internet isgenerally known to those skilled in the art and, as such, need not bedetailed herein.

“Cloud” or “Cloud computing” includes a model for enabling convenient,on-demand network access to a shared pool of configurable computingresources (e.g., networks, servers, storage, applications, and services)that can be rapidly provisioned and released with minimal managementeffort or service provider interaction. Cloud computing may includelocation-independent computing, whereby shared servers provideresources, software, and data to computers and other devices on demand.

Systems, methods and computer program products are provided. Referencesto “various embodiments”, in “some embodiments”, “one embodiment”, “anembodiment”, “an example embodiment”, and the like, indicate that theembodiment described may include a particular feature, structure, orcharacteristic, but every embodiment may not necessarily include theparticular feature, structure, or characteristic. Moreover, such phrasesare not necessarily referring to the same embodiment. Further, when aparticular feature, structure, or characteristic is described inconnection with an embodiment, it is submitted that it is within theknowledge of one skilled in the art to affect such feature, structure,or characteristic in connection with other embodiments whether or notexplicitly described. After reading the description, it will be apparentto one skilled in the relevant art(s) how to implement the disclosure inalternative embodiments.

The steps of a method or algorithm described in connection with theexamples disclosed herein may be embodied directly in hardware, in asoftware module executed by the processor 102, or in a combination ofthe two. A software module may reside in RAM memory, flash memory, ROMmemory, EPROM memory, EEPROM memory, registers, hard disk, a removabledisk, a CD-ROM, or any other form of storage medium known in the art. Anexemplary storage medium is coupled to the processor 102 such that theprocessor 102 can read information from, and write information to, thestorage medium. In the alternative, the storage medium may be integralto the processor 102. The processor 102 and the storage medium mayreside in an Application Specific Integrated Circuit (ASIC).

The methods/systems 100-1100 may be described herein in terms offunctional block components, screen shots, optional selections andvarious processing steps. It should be appreciated that such functionalblocks may be realized by any number of hardware and/or softwarecomponents configured to perform the specified functions. For example,the methods/systems 100-1100 may employ various integrated circuitcomponents, e.g., memory elements, processing elements, logic elements,look-up tables, and the like, which may carry out a variety of functionsunder the control of one or more microprocessors or other controldevices. Similarly, the software elements of the methods/systems100-1100 may be implemented with any programming or scripting languagesuch as, VPL, C, C++, C#, Java, JavaScript, VBScript, Macromedia ColdFusion, COBOL, Microsoft Active Server Pages, assembly, PERL, PHP, awk,Python, Visual Basic, SQL Stored Procedures, PL/SQL, any UNIX shellscript, and extensible markup language (XML) with the various algorithmsbeing implemented with any combination of data structures, objects,processes, routines or other programming elements. Further, it should benoted that the methods/systems 100-1100 may employ any number ofconventional techniques for data transmission, signaling, dataprocessing, network control, and the like.

As will be appreciated by one of ordinary skill in the art, themethods/systems 100-1100 may be embodied as a customization of anexisting system, an add-on product, upgraded software, a stand-alonesystem, a distributed system, a method, a data processing system, adevice for data processing, and/or a computer program product.Furthermore, the methods/systems 100-1100 may take the form of acomputer program product on a non-transitory computer-readable storagemedium having computer-readable program code means embodied in thestorage medium. Any suitable computer-readable storage medium may beutilized, including hard disks, CD-ROM, optical storage devices,magnetic storage devices, and/or the like.

Exemplary embodiments of the methods/systems 100-1100 have beendisclosed in an illustrative style. Accordingly, the terminologyemployed throughout should be read in a non-limiting manner. Althoughminor modifications to the teachings herein will occur to those wellversed in the art, it shall be understood that what is intended to becircumscribed within the scope of the patent warranted hereon are allsuch embodiments that reasonably fall within the scope of theadvancement to the art hereby contributed, and that that scope shall notbe restricted, except in light of the appended claims and theirequivalents.

What is claimed is:
 1. A computer-based method comprising the steps of:detecting, using at least one sensor, a data associated with at leastone person, an object, a living being, a place, an environment, orcombinations thereof within a proximity of at least one of the at leastone sensor; selectively storing, using a processor, a learned data in adatabase based on the detected data and a pre-programmed algorithmstored in the database; passively outputting, using a communicationsunit or an output device coupled to or in communication with theprocessor, a first output data upon a request received by the processoror a predetermined or scheduled event stored in the database; andactively and automatically outputting, using the communications unit orthe output device, a second output data based on the detected data, thelearned data, and the pre-programmed algorithm.
 2. The method of claim1, wherein the detected data corresponds to at least one of: a visualcharacteristic, a speech, tactile information, a motion, an interaction,an action, a behavior, a conversation, a location, an identity, anattribute, or combinations thereof associated with the at least oneperson, a motion, a sound, an interaction, an action, tactileinformation, a visual characteristic, a location, an identity, orcombinations thereof associated with the object, the living being, theplace, the environment, or combinations thereof within a proximity of atleast one of the at least one sensor, or a visual characteristic, atype, a location, a temperature, a moisture content, a weathercondition, a sound, an attribute, or combinations thereof associatedwith the place or the environment within a proximity of at least one ofthe at least one sensor.
 3. The method of claim 1, further comprising:analyzing, using the processor, the detected data based on the learneddata corresponding to the at least one person, the object, the livingbeing, the event, the place, the environment, or combinations thereof,wherein the steps of selectively storing the database and actively andautomatically outputting the second output data are further based on theanalyzed data.
 4. The method of claim 1, wherein the method isimplemented in a vehicle.
 5. The method of claim 1, wherein the methodis implemented in a portable electronic device.
 6. The method of claim 1further comprising: analyzing, using the processor, a conversation or aninteraction based on the detected data, wherein the step of actively andautomatically outputting the second output data further includes atleast one of: actively and automatically generating, using thecommunications unit or the output device, an output speech, an outputimage, an output video, or combinations thereof for actively andautomatically initiating or participating in the conversation or theinteraction with the at least one person, or actively and automaticallygenerating, using the communications unit or the output device, theoutput speech, the output image, the output video, or combinationsthereof for actively communicating a reminder to the user.
 7. The methodof claim 1 further comprising: determining, using the processor, arelationship, a sequence, or a correlation between at least two of theat the at least one person, the object, the living being, the event, theplace, or the environment based on the learned data and thepre-programmed algorithm, wherein the step of actively and automaticallyoutputting the second output data is further based on the determinedrelationship, sequence, or correlation.
 8. The method of claim 1,further comprising: actively and automatically improving a quality of orincreasing a quantity of the detected data by performing at least oneof: actively and automatically outputting, using the communications unitor the output device, the second output data in form of a question or aninquiry for eliciting a response or an input from the at least oneperson, or actively and automatically outputting, using thecommunications unit or the output device, the second output data forparticipating in a conversation or an interaction with the at least oneperson.
 9. The method of claim 1, wherein: the at least one sensor is atleast one of an image sensor, a vibration sensor, a microphone, aproximity detector, a position detector, a force sensor, a distancesensor, a laser scanner, or a radar sensor, and the at least one sensoris integrated in or coupled to at least a vehicle or a portableelectronic device.
 10. The method of claim 1, wherein the step ofactively and automatically outputting the second output data includes atleast one of: displaying, using a display coupled to or in communicationwith the communications unit or the output device, an output image or anoutput video, or generating, using a speaker coupled to or incommunication with the communications unit or the output device, anoutput speech.
 11. The method of claim 1, wherein the database includesor is in communication with a cloud-based database, the processor beingfurther configured to: transmit, using the processor and the cloud-baseddatabase, the detected data, the first output data, the second outputdata, or combinations thereof to an external device coupled to or incommunication with the cloud-based database, and receive, using theprocessor and the cloud-based database, an input data from the externaldevice.
 12. The method of claim 1, wherein the at least one personincludes a user with disabilities or suffering from health or memorycomplications, and wherein the step of actively and automaticallyoutputting the second output data further includes at least one of:associating the detected data with the learned data corresponding to theperson, the object, the living being, the event, the place, theenvironment, or combinations thereof, wherein the second output datacorresponds to a reminder data directed to communicating information tothe user with respect to identification, characteristics, location,actions, or interactions of the at least one person, the object, theliving being, the event, the place, the environment, or combinationsthereof.
 13. The method of claim 1, wherein the method furthercomprises: analyzing, using the processor, health, exercise, or dietactivities of the at least one person based on the detected data,wherein the step of actively and automatically outputting the secondoutput data includes at least one of: outputting, using thecommunications unit or the output device, the second output data to ahealth professional for assisting the health professional in evaluationof the health, exercise, or diet activities of the at least one person,or outputting, using the communications unit or the output device, thesecond output data to the at least one person for outputting arecommendation associated with the health, exercise, or diet activities.14. The method of claim 1, wherein the pre-programmed algorithmcorresponds to appropriateness algorithm rules for determining:information content of the second output data, privacy concerns of auser, context of a detected event, or combinations thereof, the methodfurther comprising: analyzing, using the processor, the detected databased on the appropriateness algorithm rules and the learned data,wherein the step of actively and automatically outputting the secondoutput data is based on the analyzed data.
 15. A vehicle comprising: atleast one sensor configured to detect a data associated with a driver, apassenger, an object, a living being, an event, a place, an environment,or combinations thereof within a proximity of at least one of the atleast one sensor; a database storing a pre-programmed algorithm; acommunications unit; and a processor coupled to or in communication withthe at least one sensor, the database, and the communications unit, theprocessor being configured to: selectively store a learned data in thedatabase based on the detected data and the pre-programmed algorithm;analyze the detected data based on the learned data and thepre-programmed algorithm; passively output, using the communicationsunit, a first output data upon a request received by the processor or apredetermined or scheduled event stored in the database; and activelyand automatically output, using the communications unit, a second outputdata based on the analyzed data, the learned data, and thepre-programmed algorithm.
 16. The vehicle of claim 15, wherein thecommunications unit includes at least a display, a microphone, and aspeaker, the processor being further configured to: detect, using aspeech recognition algorithm, a speech of at least one of the driver orthe passenger, analyze, using a speech processing algorithm, thedetected speech based on the learned data and the pre-programmedalgorithm, determine a relationship, a sequence, or a correlationbetween the detected speech and the driver, the passenger, the object,the living being, the event, the place, the environment, or combinationsthereof, and actively and automatically perform at least one of:generating, using the speaker, an output speech based on the determinedrelationship, sequence, or correlation, or displaying, using thedisplay, an output image or an output video based on the determinedrelationship, sequence, or correlation.
 17. The vehicle of claim 16,further comprising: determining, using the processor, a current expectedlevel of driving attention based on the detected data and based on acurrent operation of the vehicle; determining, using the processor, anoutput timing and an information content of the generated output speechor the displayed output image or output video based on the determinedcurrent expected level of driving attention.
 18. The vehicle of claim17, wherein the database includes or is in communication with acloud-based database, the processor being further configured to:transmit, using the processor and the cloud-based database, the detecteddata, the first output data, the second output data, or combinationsthereof to an external device, and receive, using the processor and thecloud-based database, an input data from the external device.
 19. Aportable electronic device comprising: at least one sensor configured todetect a data associated with at least one person or a user, an object,a living being, an event, a place, an environment, or combinationsthereof within a proximity of at least one of the at least one sensor; adatabase storing a pre-programmed algorithm; a communications unit; anda processor coupled to or in communication with the at least one sensor,the database, and the communications unit, the processor beingconfigured to: selectively store a learned data in the database based onthe detected data and the pre-programmed algorithm; analyze the detecteddata based on the learned data and the pre-programmed algorithm;passively output, using the communications unit, a first output dataupon a request received by the processor or a predetermined or scheduledevent stored in the database; and actively and automatically output,using the communications unit, a second output data based on theanalyzed data, the learned data, and the pre-programmed algorithm. 20.The portable electronic device of claim 19, wherein the communicationsunit includes at least a display, a microphone, and a speaker, theprocessor being further configured to: detect, using a speechrecognition algorithm, a speech of the at least one person, analyze,using a speech processing algorithm, the detected speech based on thelearned data and the pre-programmed algorithm, determine a relationship,a sequence, or a correlation between the detected speech and the learneddata, and actively and automatically perform at least one of:generating, using the speaker, an output speech based on the determinedrelationship, sequence, or correlation, or displaying, using thedisplay, an output image or an output video based on the determinedrelationship, sequence, or correlation.